Field of the Invention
The present application is related to automated techniques for evaluating work product and, in particular, to techniques that employ feature extraction and machine learning to efficiently and consistently evaluate instances of media content that constitute, or are derived from, coursework submissions.
Description of the Related Art
As educational institutions seek to serve a broader range of students and student situations, on-line courses have become an increasingly important offering. Indeed, numerous instances of an increasingly popular genre of on-line courses, known as Massive Open Online Courses (MOOCs), are being created and offered by many universities, as diverse as Stanford, Princeton, Arizona State University, the Berkeley College of Music, and the California Institute for the Arts. These courses can attract hundreds of thousands of students each. In some cases, courses are offered free of charge. In some cases, new educational business models are being developed, including models in which students may be charged for deeper evaluation and/or credit, or in which advertising provides a revenue stream.
While some universities have created their own Learning Management Systems (LMS), a number of new companies have begun organizing and offering courses in partnership with universities or individuals. Examples of these include Coursera, Udacity, and edX. Still other companies, such as Moodle and Blackboard, offer LMS designs and services for universities who wish to offer their own courses.
Students taking on-line courses usually watch video lectures, engage in blog/chat interactions, and submit assignments, exercises, and exams. Submissions may be evaluated (to lesser or greater degrees, depending on the type of course and nature of the material), and feedback on quality of coursework submissions can be provided. While many courses are offered that evaluate submitted assignments and exercises, the nature and mechanics of the evaluations are generally of four basic types:                1) In some cases, human graders evaluate the exercises, assignments, and exams. This approach is labor intensive, scales poorly, can have consistency/fairness problems and, as a general proposition, is only practical for smaller online courses, or courses where the students are (or someone is) paying enough to hire and train the necessary number of experts to do the grading.        2) In some cases, assignments and exams are crafted in multiple-choice, true false, or fill-in-the blank style, such that grading by machine can be easily accomplished. In some cases, the grading can be instant and interactive, helping students learn as they are evaluated, and possibly shortening the exam time, e.g., by guiding students to harder/easier questions based on responses. However, many types of subject matter, particularly those in which artistic expression or authorship are involved, do not lend themselves to such assignment or examination formats.        3) In some cases, researchers have developed techniques by which essay-style assignments and/or exams may be scanned looking for keywords, structure, etc. Unfortunately, solutions of this type are, in general, highly dependent on the subject matter, the manner in which the tests/assignment are crafted, and how responses are bounded.        4) In some cases, peer-grading or assessment has been used, whereby a student is obligated to grade the work of N other students. Limitations of, and indeed complaints with, peer-assessment include lack of reliability, expertise and/or consistency. Additional issues include malicious or spiteful grading, general laziness of some students, drop-outs and the need to have students submit assignments at the same time, rather than at individual paces.        
Improved techniques are desired, particularly techniques that are scalable to efficiently and consistently serve large student communities and techniques that may be employed in subject matter areas, such as artistic expression, creative content computer programming and even signal processing, that have not, to-date, proved to be particularly amenable to conventional machine grading techniques.